TD Control: Sarsa

Monte Carlo (MC) control methods require us to complete an entire episode of interaction before updating the Q-table. Temporal Difference (TD) methods will instead update the Q-table after every time step.

## Video

Watch the next video to learn about Sarsa (or Sarsa(0) ), one method for TD control.

## Video

## Pseudocode

In the algorithm, the number of episodes the agent collects is equal to num_episodes . For every time step t\geq 0 , the agent:

• takes the action A_t (from the current state S_t ) that is \epsilon -greedy with respect to the Q-table,
• receives the reward R_{t+1} and next state S_{t+1} ,
• chooses the next action A_{t+1} (from the next state S_{t+1} ) that is \epsilon -greedy with respect to the Q-table,
• uses the information in the tuple ( S_t , A_t , R_{t+1} , S_{t+1} , A_{t+1} ) to update the entry Q(S_t, A_t) in the Q-table corresponding to the current state S_t and the action A_t .